Average Case Analysis of the Classical Algorithm for Markov Decision Processes with B\"uchi Objectives

We consider Markov decision processes (MDPs) with $\omega$-regular specifications given as parity objectives. We consider the problem of computing the set of almost-sure winning vertices from where the objective can be ensured with probability 1. The algorithms for the computation of the almost-sure winning set for parity objectives iteratively use the solutions for the almost-sure winning set for B\"uchi objectives (a special case of parity objectives). We study for the first time the average case complexity of the classical algorithm for computing almost-sure winning vertices for MDPs with B\"uchi objectives. Our contributions are as follows: First, we show that for MDPs with constant out-degree the expected number of iterations is at most logarithmic and the average case running time is linear (as compared to the worst case linear number of iterations and quadratic time complexity). Second, we show that for general MDPs the expected number of iterations is constant and the average case running time is linear (again as compared to the worst case linear number of iterations and quadratic time complexity). Finally we also show that given all graphs are equally likely, the probability that the classical algorithm requires more than constant number of iterations is exponentially small

Symbolic Algorithms for Qualitative Analysis of Markov Decision Processes with B\"uchi Objectives

We consider Markov decision processes (MDPs) with \omega-regular specifications given as parity objectives. We consider the problem of computing the set of almost-sure winning states from where the objective can be ensured with probability 1. The algorithms for the computation of the almost-sure winning set for parity objectives iteratively use the solutions for the almost-sure winning set for B\"uchi objectives (a special case of parity objectives). Our contributions are as follows: First, we present the first subquadratic symbolic algorithm to compute the almost-sure winning set for MDPs with B\"uchi objectives; our algorithm takes O(n \sqrt{m}) symbolic steps as compared to the previous known algorithm that takes O(n^2) symbolic steps, where $n$ is the number of states and $m$ is the number of edges of the MDP. In practice MDPs have constant out-degree, and then our symbolic algorithm takes O(n \sqrt{n}) symbolic steps, as compared to the previous known $O(n^2)$ symbolic steps algorithm. Second, we present a new algorithm, namely win-lose algorithm, with the following two properties: (a) the algorithm iteratively computes subsets of the almost-sure winning set and its complement, as compared to all previous algorithms that discover the almost-sure winning set upon termination; and (b) requires O(n \sqrt{K}) symbolic steps, where K is the maximal number of edges of strongly connected components (scc's) of the MDP. The win-lose algorithm requires symbolic computation of scc's. Third, we improve the algorithm for symbolic scc computation; the previous known algorithm takes linear symbolic steps, and our new algorithm improves the constants associated with the linear number of steps. In the worst case the previous known algorithm takes 5n symbolic steps, whereas our new algorithm takes 4n symbolic steps

Peripheral Blood Mononuclear Cell Cytokine and Proliferative Response to In Vitro Echinacea Stimulaton in Male College Wrestlers andSoccer Players During Preseason Practice

Advisor: David S. SenchinaThe effects of dietary botanical supplements on the immune response in athletes are unknown, despite a recent increase in herbal supplement use by both college and professional athletes. We conducted 2 separate studies to examine the effects of in vitro Echinacea stimulation on the immune responses of peripheral blood mononuclear cells (PBMCs) taken from athletes during preseason training. College-level male competitive athletes from 2 sports (wrestling and soccer) participated in the studies. PBMCs were isolated from blood sampled either pre- or post-practice, standardized to the same concentration, and then stimulated with extracts from Echinacea pallida, Echinacea simulata, or solvent vehicle control. Cytokine production (TNF, IL-Ib, IL-I0, and IFN-g) was measured from super-natants collected between 24-72 hrs contingent on the specific cytokine; proliferation was assessed at 72 hrs. Extracts were phytochemically profiled by high pressure liquid chromatography to quantify known bioactive compounds including alkamides and caffeic acid derivatives. Results differed between the wrestlers and soccer players. In general, E. simulata was a more potent immunomodulator than E. pallida in both studies. Following exercise, PBMC production of TNF, IL-l0, and IFN-g production either decreased or was unaffected. IL-lb levels showed no change in either study. PBMC proliferation increased in the wrestlers as a result of training, but decreased in the soccer players. In conclusion, observed effects were contingent on species chosen, time point within preseason training, and sport (training type).Drake University, College of Pharmacy and Health Sciences, Department of Biolog

LIPIcs

We consider Markov decision processes (MDPs) with specifications given as Büchi (liveness) objectives. We consider the problem of computing the set of almost-sure winning vertices from where the objective can be ensured with probability 1. We study for the first time the average case complexity of the classical algorithm for computing the set of almost-sure winning vertices for MDPs with Büchi objectives. Our contributions are as follows: First, we show that for MDPs with constant out-degree the expected number of iterations is at most logarithmic and the average case running time is linear (as compared to the worst case linear number of iterations and quadratic time complexity). Second, for the average case analysis over all MDPs we show that the expected number of iterations is constant and the average case running time is linear (again as compared to the worst case linear number of iterations and quadratic time complexity). Finally we also show that given that all MDPs are equally likely, the probability that the classical algorithm requires more than constant number of iterations is exponentially small